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SYSTEMS AND METHODS FOR INTRAOPERATIVE TARGETING 

BACKGROUND 

In recent years, the medical community has been increasingly focused on minimizing 
the invasiveness of surgical procedures. Advances in imaging technology and instrumentation 
5 have enabled procedures using minimally-invasive surgery with very small incisions. Growth 
in this category is being driven by a reduction in morbidity relative to traditional open 
procedures, because the smaller incisions minimize damage to healthy tissue, reduce patient 
pain, and speed patient recovery. The introduction of miniature CCD cameras and their 
associated micro-electronics has broadened the application of endoscopy from an occasional 
10 biopsy to full minimally-invasive surgical ablation and aspiration. 

Minimally-invasive endoscopic surgery offers advantages of a reduced likelihood of 
intraoperative and post-operative complications, less pain, and faster patient recovery. 
However, the small field of view, the lack of orientation cues, and the presence of blood and 
obscuring tissues combine to make video endoscopic procedures in general disorienting arid 
15 challenging to perform. Modern volumetric surgical navigation techniques have promised 
better exposure and orientation for minimally-invasive procedures, but the effective use of 
current surgical navigation techniques for soft tissue endoscopy is still hampered by two 
difficulties: (1) accurately tracking all six degrees of freedom (DOF) on a flexible endoscope 
within the body, and (2) compensating for tissue deformations and target movements during 
20 an interventional procedure. 

To illustrate, when using an endoscope, the surgeon's vision is limited to the camera's' 
narrow field of view and the lens is often obstructed by blood or fog, resulting in the surgeon 
suffering a loss of orientation*. Moreover, endoscopes can display only visible surfaces and it 
is therefore often difficult to visualize tumors, vessels; and other anatomical structures that lie 
25 beneath opaque tissue (e.g., targeting of pancreatic adenocarcinomas via gastro-intestinal 
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endoscopy, or feting of Mucosa! lesions to sample peri-inline, such as 

masses in the liver, or targeting of sublominal lesion in (he bronehi). 

Recently, image-guided therapy (IOT) <— tatrodUCe4 Tta - ^ 

co»p,emen, conventional endoseopy and have heen used predominantly in neuro.ogica!, 
5 sinus, and spinal surgery, where hony or marker-based registration can provide adequate 
^e, accuracy using preoperative images (typically 1-3 nun). While IOT enhances me 
surgeon's ability to direct instruments andtarge, specific anatomical sfruchnes, in soft tissue 
mese systems .a* sufficient targeting accuracy due to infra-operative tissue movement and 
deformation. In addition, since an endoscope provides a video representation of a 3D 
,0 environment, i, is difficult to correlate the conventional, purely 2D IOT images with the 
endoscope video. Correlation of information obtained from inoperative 3D mtrasonic 
imaging with video endoscopy can significantly improve the accuracy o, localization and 
targeting in minimally-invasive K3T procedures. 

Until die mid 1990% the most common use of image guidance was for stereotactic 
« biopsies, in which a surgical trajectory device and a frame of reference * used. Traditional 
firnne-basedmelhodsof ataxia definedthe intracranial anatomy with reference to a set of 
fiducial markers, which were attached to a frame that was screwed hue the patient, skull. 
These fiducials were measured on pre-operative tomographic (MRIorCT) images. 

A trajectory-enforcement device was placed on top of the frame of reference and used 
20 ,o guide tire biopsy tool to ore targe, lesion, baaed on prior calculations obtained from pre- 
operative data. The use of a mechanical frame allowed for high localization accuracy, bu, 
eaused patient discomfort, limited surgical flexibility, and did not a„ow me surgeon * 
visualize the approach of me biopsy tool to the lesion. 

There has been a gradual emergence of image guided technics mat eliminate the 
25 need for the frame altogether. The flrs, frameless stereotactic system used - articulated 



robotic arm to register preoperative imaging with me patient's anatomy in the operating 
room. This was followed by the use of acoustic devices for necking instruments in the 
operating environment The acoustic devices evenboally were superceded by optica, .racking 
syst ems, which use a camera and infrared diodes (or reflect) atmched to a moving object 
5 ,o accurately track its position and orientation. These aysfcms use markers placed exmmally 
on me patient to register preoperative imaging witir the patient's anatomy in the operating 
room. Such intra-operative navigation techniques use preoperative CT or MR images to 
p^vide localized information during surgery. In addition, all systems enhance intia-operative 
.ocalization by providing feedback regarding the location of .he surgical u~«s with 

10 respect to 2D preoperative data. 

Unti! recently, volumetric surgical navigation has been Umi.ee by the lack of the 
computational power require* to produce real-time 3D images. The use of various 
volumetric imaging motilities has progressed to permit the physician to visualize and 
quantify the extent o, disease in 3D in order to plan and execute tieatmen,. Systems are 
15 curtently able to provide reel-time nrsion of pre-operative 3D dam with intraoperative 2D 
oata images from video cameras, oUrasound probes, aurgieal microscopes, and endoscopes. 
These systems have been used predominantly in neurological, sines, and spina, surgery, 
where direct access to the preoperative data plays a major ro.e in the execution o, me 
surgica. task. This is despite the ft* that, because of movement and deformation of me tissue 
20 during me surgery, mese IGT procedures mud to lose men spatia, registration wim respeet » 
the pre-operatively acquired image. 
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SUMMARY 

The metod of some embodiment of to invention assists a user in Riding a medical 
_n, to a — ee target site in a patient. This method generates one or more 
^operative images. The method indicates a target site on the image© The method 
5 ermines 3-D coordinates of the target site in a reference coordinate system. The method 
(1) «*, the position of the h_, in me reference coordhrate system, (2) projects onto 
aoUplaydeviceaviewfieUl as seen fiomto position witi, respect to me tirol in me reference 
coordinate system, and (3) projects onto the displayed view field indicia of the targe, si. 
corresponding «o the position, ht some emhodiments, me field of view is a view no, on!y 
10 from me position of me h— bo, aUo from a .mown orien«a«ion of me instiumen, in to 
Terence coordinate system. By ohaerving me indicia, to nser can .pride ,he in— 
toward me Urge, ai,e hy moving to insnumen, so ma, to indicia are piaced or held m a 

given state in the displayed field of view. 

ln some embodiments, ,he metod generates intraoperative image© by using an 
15 oUrasonic sonroe ,o generate an ultrasonic image of tire patien,. In some of ,hese 
embodiment the 3-D coordinates of a spatia, target site indicated on me image are 
to ined from tire 2-D coordma.es of me spatia, targe, site on tire image and me position of 
the ultrasonic source. 

In some embodiment me medical insuumen, is an endoscope and to view fie,d 
2 „ projected on,o to display device can be the image seen by to endoscope. Tne view field 
ejected on,o to display device can be to view to, is seen from to tip^nd position and 
orientation of to mediea, instmmen, having a defined field of view. The view field 
projected onte to display device can be to view to, is seen from a position along to axra 
of instrument to, is different from to target than to tip-end position of to medical 
25 instrument. 



The spatia. target site indicated ean he a voiume, area, or point. In acme 
stents, the indicia are arranged h a geonrehic patten, that defines fire houndary of the 
Seated spafiai feature or the position of a point within the target site. The spacing hetween 
or antong indicia can he indicative of the distance of *. h_, frotn the target-atte 
5 position. The siae or shape of the individua, indicia can indicate fit. distance of the 
instrument from the target-site position. 

Toe size or shape of individua. indicia can aiso he indicative of the orientation of the 
t00 , Por instance, the indicia cou.d provide on each hnage a second spafia, feature that 
toget „er with a. first-indicated spatiai feahrre, defines a sur^ca, u*-, on the dispiayed 
10 im age. On a patient sor.ce tegion, dte — can indicate an entry point .at defines 

Rectory on the dispiayed hnage can he indicated hy two sets of indicia, a firs, ae, 
responding ,0 the vindicated spatia, fea.ro and a second set corresponding to the 
^ndspatiaifeahneortheindicatedentrypoint The snrgica, Rectory on fit. disp.ayed 

sp afia! feature and the second spafia! feahrre or indicated entry point 

Score enshodinrenta provide a aysfcn, that guidea a nredicai in— to a targe. ,te 
in . p atien, This aysten, induces an inraging device for generafing one or more 
^operative onagea on which a pafient target site can he defined in a three—ona, 
20 ooordinatcsyatenr.Thiaay—inciudes 

«. inciudes an indicator hy which a user can indicate a spafiat target site on such 

, i , Awir* as well as an electronic computer 
image(s). The system also includes a (hsplay devxce, as well 

operably connected to me tracking system, display device, and indicator. 
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Finally, this system includes a computer-readable code that directs the computer (i) to 
record target-site spatial information indicated by the user on said image(s), through the use 
of said indicator, (ii) to determine from the spatial target site indicated on said image(s), 3-D 
coordinates of the spatial target site in a reference coordinate system, (iii) to track the 
5 position of the instrument in the reference coordinate system, (iv) to project onto a display 
device a view field as seen from a known position with respect to the tool in the reference 
coordinate system, and (v) to projects onto the displayed view field indicia of the spatial 
target site corresponding to the predetermined position. In some embodiments, the field of 
view is a view that is not only from the position of the instrument but also from a known 
10 orientation of the instrument in the reference coordinate system. By observing the indicia, the 
user can guide the instrument toward the target site by moving the instrument so that the 
indicia are placed or held in a given state in the displayed field of view. 

In some embodiments, the imaging device is an ultrasonic imaging device capable of 
generating digitized images of the patient target site from any position, respectively. In 
15 addition, in some embodiments, the tracking device is operable to record the positions of the 
imaging device at the two positions. Also, in some embodiments, the medical instrument is 
an endoscope and the view field projected onto the display device is the image seen by the 
endoscope. 

Some embodiments of the invention provide a machine readable code in a system that 
20 is designed to assist a user in guiding a medical instrument to a target site in a patient. The 
system includes (a) an imaging device for generating one or more intraoperative images on 
which a patient target site can be defined in a three-dimensional coordinate system, (b) a 
tracking system for tracking the position of the medical instrument and imaging device in a 
reference coordinate system, (c) an indicator by which a user can indicate a spatial target site 
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on such image*), W a dispiay device, and (e) an decbonic compumr opembiy conned «o 
the tracking system, display device, and indicator. 

The code inc.uding se* of inactions for (i) recording .arget-stte spatia. information 
boated by the user on the image(s, tinough me use ofthe indicator, (U) determimng from 
, me spatia, to ge, site indicated on said ima g e(s), 3-D coordinates ofme spatia. target site in a 
Terence coordinate system, (Hi) tracking me position of me instrument in the reference 
^inate system, (iv) projecting onto a dis pl ay device, a view fie,d as seen from a known 
position with respect to me mo. in me reference coordinate system, and (v) projecting onto 
to dispUyed view fie,d, indicia for indicating the indicated spatial target site with respect to 
10 to known position, .n some embodiments, the fie,d of view is a view no, only from me 
posW on of the instrument hut also from a known orientation of me insbmmen, in me 
reference coordinate system. By observing the indicia, the user can guide the indent 
mward me mrge, sne by moving me in— so that me indicia are placed or held m a 
given state in (he displayed field of view. 

Some embodiments provide a method that assists a user in guiding a med.ca, 
inst rumen, to a subsurface target site in a patient The method (1) indicates a spatia! target 
sit e on an intraoperative image ofme patient, (2) determines duee-dimensional cooroina.es of 
the targe, aim in a reference coordinate sys«em, (3) determines a position ofme u_. in 
.» reference coordinate system, (4) projee* onto a disp,ay device a view field from a 
20 predetermined position reiative * me ins«rumen, m me reference coordinate system, and 
(5) projects onto the view field an indicia of me spatia, targe, site copending to the 
predetermined position. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The novel features of the invention are set forth in the appended claims. However, for 
purpose of explanation, several embodiments of the invention are set forth in the following 
figures. 

Figs. 1-2 show exemplary flowcharts of the operation of the system of some 
embodiments of the invention. 

Figs. 3-4 shows exemplary user interface displays of the system of some 
embodiments of the invention. 

Figs. 5-6 shows exemplary operating set-up arrangements in accordance with one 

aspect of the system. 
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DETAILED DESCRIPTION OTP TTTF, INVENTION 



In the following description, numerous details are set forth for purpose of explanation. 
However, one of ordinary skill in the art will realize that the invention may be practiced 
5 without the use of these specific details. In other instances, well-known structures and 
devices are shown in block diagram form in order not to obscure the description of the 
invention with unnecessary detail. 

Figure 1 illustrates a process 100 of some embodiments of the invention. This process 
guides a medical instrument to a desired position in a patient. As shown in this figure, the 
10 process 100 initially acquires (at 105) one or more intraoperative images of the target site. 
Next, the process 100 registers (at 110) the intraoperative images, the patient target site, and 
the surgical instruments into a common coordinate system. 

The patient, the imaging source(s) responsible for the intraoperative images and 
surgical tool must all be placed in the same frame of reference (in registration). This can be 
15 done by a variety of methods, three of which are described below. First, a wall-mounted 
tracking device can be used to track the patient, imaging source(s), and the surgical tool (e.g., 
endoscope). Second, only the position of the tool can be tracked. Under such an approach, the 
tool can be placed in registration with the patient and imaging source by touching the tool 
point to fiducials on the body and to the positions of the imaging source(s). Thereafter, if the 
20 patient moves, the device could be registered by tool-to-patient contacts. That is, once the 
images are made, from known coordinates, it is no longer necessary to further track the 
position of the image source(s). 

Third the patient and image sources are placed in registration by fiducials on the 
patient and in the images, or alternatively, by placing the imaging device at known 
25 coordinates with respect to the patient. The patient and tool are placed in registration by 
detecting the positions of fiducials with respect to the tool, e.g., by using a detector on the 
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tool for detecting the positions of the patient fiducials. Alternatively, the patient and an 
endoscope tool can be placed in registration by imaging the fiducials in the endoscope, and 
matching the imaged positions with the position of the endoscope. 

After the registration operation at 110, the process 100 tracks (at 1 15) the position of 
the surgical instrument with respect to the patient target site. In some embodiments, a 
magnetic tracking system is used to track the endoscope for navigation integration in one 
implementation. The system provides a magnetic transducer into the working channel at the 
endoscope tip, positioning the field generator so that the optimal sensing volume 
encompasses the range of sensor positions. In one implementation that provides for six 
degrees of freedom (6 DOF), a miniaturized magnetic tracking system with metal 
insensitivity can be used. The tracking system may be calibrated using a calibration jig. A 
calibration target is modified from a uniform to a non-uniform grid of points by reverse- 
mapping the perspective transform, so that the calibration target point density is 
approximately equal throughout the endoscope image. The calibration jig is waterproofed 
and designed to operate in a submerged environment. Where appropriate, calibration will be 
performed while the jig is immersed in a liquid with refractive properties similar to the 
operating environment. 

In one embodiment, an ultrasound calibration system can be used for accurate 
reconstruction of volumetric ultrasound data. An optical tracking system is used to measure 
the position and orientation of a tracking device that will be attached to the ultrasound probe. 
A spatial calibration of intrinsic and extrinsic parameters of the ultrasound probe is 
performed. These parameters are used to transform the ultrasound image into the co-ordinate 
frame of the endoscope's field of view. In another embodiment, a magnetic tracking system 
is used for the ultrasound probe. Using only one tracking system for both the endoscope and 
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the ultrasound probe reduces obstructions in the environment, and avoids a line-of-sight 
tracking requirement 

In another embodiment, tracking of the probe is done using an optical tracking 
system. The calibration of the 3D probe is done in a manner similar to a 2D ultrasound probe 

5 calibration using intensity-based registration. Intensity-based registration is fully automatic 
and does not require segmentation or feature identification. In the typical 2D case, acquired 
images are subject to scaling in the video generation and capture process. This 
transformation and the known position of the tracking ultrasonic calibration device 
(calibration phantom) are used to determine the relationship between the ultrasound imaging 

10 volume and the ultrasound probe's tracking device. Successful calibration requires an 
unchanged geometry. The calibration phantom will be designed to withstand relocation and 
handling without deformation. A quick-release clamp attached to the phantom will hold the 
ultrasound probe during the calibration process. 

A spatial correlation of the endoscopic video with dynamic ultrasound images is then 

15 done. The processing internal to each tracking system, endoscope, and ultrasound machine 
causes a unique time delay between the real-time input and output of each device. The output 
data streams are not synchronized and are refreshed at different intervals. In addition, the 
time taken by the navigation system to acquire and process these outputs is stream- 
dependant. Consequently, motion due to breathing and other actions can combine with these 

20 independent latencies to cause real-time display of dynamic device positions different to 
those when the imaging is actually being acquired. 

In some embodiments, a computer is used to perform the spatial correlation. The 
computer can handle a larger image volume, allowing for increased size of the physical 
imaged volume or higher image resolution (up to 5 12 _ 512 _ 5 12 instead of 256 _ 256 _ 64). 

25 The computer also provides faster 3D reconstruction and merging, and a higher-quality 
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perspective volume rendering at a higher frame rate. The computer time-stamps and buffers 
the tracking and data streams, and then interpolating tracked device position and orientation 
to match the image data timestamps. 

In determining the required time offset, the ultrasound probe is moved across a step 
5 surface in the calibration phantom to create a temporal step function in both the tracking 
system and image data stream. The relative delay is determined by comparing the timestamps 
of the observed step function in each data stream. The endoscope latency is determined 
similarly using the same phantom. In some embodiments, this is done whenever the 
ultrasound system is reconfigured. The endoscope latency will not need to be recalculated 
10 unless the endoscope electronics are changed, however. The patient is imaged through the 
ultrasound probe, and the endoscope becomes the frame of reference for the surgeon. The 
important information is contained in the dynamic relationship of the ultrasound data to the 
endoscope video, which is known through calibration and tracking of both devices. 

Turning now to Figure 1, the process shows (at 120) on a display device one or more 
15 images of the patient target site. Next, the process receives (at 125) a user's indication of a 
spatial feature of the patient target site on the images of the patient target site. The process 
then projects (at 130) an indicia on the images relating the position and orientation of the 
surgical instruments to the spatial feature of the patient target site. 

The methodology illustrated in Figure 1 dynamically tracks and targets lesions in 
20 motion beyond the visible endoscopic view. When a target is identified, the subregion 
surrounding the target in the ultrasound volume will be stored as a reference, together with 
the tracked orientation of the volume. A subregion of each successively-acquired ultrasound 
volume, centered at the target position in the preceding volume, will be re-sampled using the 
orientation of the reference target subregion. Three-dimensional cross-correlation of the re- 
25 sampled subregion with the reference subregion will be used to find the new location of the 
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target. This dynamic tracking will follow each target over time; if the system is displaying 
target navigation data, the data will change in real time to follow the updated location of the 
target relative to the endoscope. 

Vascular structures return a strong, well differentiated Doppler signal. The dynamic 

5 ultrasound data may be rendered in real time using intensity-based opacity filters, making 
nonvascular structures transparent. This effectively isolates the vascular structure without 
requiring computationally-demanding deformable geometric models for segmentation, thus 
the system can follow movements and deformations in real time. 

The methodology illustrated in Figure 1 allows a user such a surgeon to mark a 

10 selected target point or region on intraoperative ultrasonic images (one or more 3-D 
ultrasound images). The designated target point or region is then displayed to the surgeon 
during a surgical operation, to guide the position and orientation of the tool toward the target 
site. In some embodiments, the target area is displayed to the user by (1) displaying a field 
representing the patient target area, and (2) using the tracked position of the tool with respect 

15 to the patient to superimpose on the field one or more indicia whose position in the displayed 
field is indicative of the relative position of the tool with respect to the marked target 
position. Also, in some embodiments, the tool is equipped with a laser pointer that directs a 
laser beam onto the patient to indicate the position and orientation of a trajectory for 
accessing the target region. The user can follow this trajectory by aligning the tool with the 

20 laser-beam. 

In the embodiments where the tool is an endoscope, the displayed image is the image 
seen by the endoscope, and the indicia are displayed on this image. The indicia may indicate 
target position as the center point of the indicia, e.g., arrows, and tool orientation for reaching 
the target from that position, by the degree of elongation of arrows, such that the indicia are 
25 brought to equal sizes when the tool is properly oriented. Alternatively, the indicia may 



indicate the surface point for entry and the elongation of the arrows, the tool orientation- 
trajectory for reaching the target from that surface point. 

Some embodiments enable surgeons to visualize a field of view of the surgical 
endoscope overlaid with volumetrically-reconstructed medical images of a localized area of 
the patients anatomy. Using this volumetric navigation system, the surgeon visualizes the 
surgical site via the surgical endoscope, while exploring the inner layers of the patient's 
anatomy through the three-dimensionally reconstructed pre-operative MRI or CT images. 
Given the endoscope's position and orientation, and given the characteristics of the camera, a 
perspective volume-rendered view matching that of the optical image obtained by the 
endoscope is rendered. This system allows the surgeon to virtually fly through and around 
the site of the surgery to visualize alternative approaches and qualitatively determine the best 
one. The volumetrically reconstructed images are generated using intensity based filtering 
and direct perspective volume rendering, which removes the need for conventional 
segmentation of high-contrast images. The real-time 3D-rendered radiographic 
reconstruction images matched with the intra-operative endoscopic images provide a new 
capability in minimally-invasive endoscopic surgery. Since hitting vascular structures 
remains the greatest hazard in endoscopic procedures, this new technology represents a 
marked improvement over conventional image-guidance systems, which generally display 
2D reconstructed images . 

In operation, and with respect to embodiments that use ultrasonic images, the user 
makes a marking on the image corresponding to the target region or site. This marking may 
be a point, line or area. From this, and by tracking the position of the tool in the patient 
coordinate system, the system functions to provide the user with visual information indicating 
the position of the target identified from the ultrasonic image. 



14 



it _ „„ e ! nn n f Fieure 1 operates in three distinct 
The navigation system that uses the process 100 ot figure P 

• a- Th* imaeed ultrasound volume will be 
modes. The first is target identification mode. The imaged u 

t arCT Pt regions of interest and mark them 
displayed to allow the surgeon to loeate one or more target regions 

for target. ^ system — —* " ^ " " " " 

m ta second mode, the endoseope win he use4 to set me position and orientation of 
to frame of reference. Based on these pa— and nsing me optiea, eharaeterlsties of me 
endoseope, the system win over,ay target navigation da, on me endosenpe video. This wd, 
anow the surgeon to target regions of mtetea, heyond me visua, range of me endoseope a 

regionsre.anvemmeendoseopenp.aaweUasapotennalrangeoferrormmisda,. 

THe third mode wi.l he used to perform the acuta, interventional procedure (such as 
Hops, or ablation) once the endoscope is in the correct position. Tne interactive imaged 
u _ votame and cross-sectiona, pianes wi,l he disced, with the .ocatlon of me 

15 ^^^^^^^^^ 
needle itself wiU also be visible in the ultrasound displays. 

The navigation system allows the interventional mo! to be positioned in me center of 
to .esion without being limited to a single, fixed 2 D ultrasound plane emananng from the 
endoscope tip. C™ 2D view capability can he duphcated by options,,, aligning a ccoss 
20 ^ona.mtrasonndp.anewimmeendosoope.) , me nrs, implementanon of, he endoscope 

inrmediatCy prior to its remove,. !n another embodiment, a sensor is integrated into the 
needle assembly, which will be in place at calibration 
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The navigation system provides real-time data on the position and orientation of the 
endoscope, and the ultrasound system provides the dynamic image data. The tip position data 
is used to calculate the location of the endoscope tip in the image volume, and the probe 
orientation data will be used to determine the rendering camera position and orientation. 

5 Surgeon feedback will be used to improve and refine the navigation system. Procedure 
durations and outcomes will be compared to those of the conventional biopsy procedure, 
performed on the phantom without navigation and image-enhanced endoscopy assistance. 

When a target is identified, some embodiments store the subregion surrounding the 
target in the ultrasound volume as a reference, together with the tracked orientation of the 

10 volume. These embodiments will then re-sample a subregion of each successively-acquired 
ultrasound volume, centered at the target position in the preceding volume, by using the 
orientation of the reference target subregion. 

Some embodiments will use three-dimensional cross-correlation of the re-sampled 
subregion with the reference subregion to find the new location of the target. This dynamic 

15 tracking will follow each target over time; if the system is displaying target navigation data, 
the data will change in real time to follow the updated location of the target relative to the 
endoscope. 

Figure 2 illustrates a process 200 of some embodiments of the invention. Like the 
process 100 of Figure 1, the process 200 guides a medical instrument to a desired position in 
20 a patient. As shown Figure 2, the process 200 initially acquires (at 205) one or more 2D or 
3D intraoperative images of the patient target site from a given orientation. Next, the process 
tracks (at 210) the position of a surgical instrument with respect to the patient target site. 

The process then registers (at 215) the intraoperative images of the patient site, the 
patient target site, and the surgical instrument into a common 3D reference coordinate 
25 system. Next, the process renders (at 220) the image of the patient target site on a display 



device. The process also specifies (at 225) a spatial feature (shape and position) of the patient 
target site on the image. The process then correlates (at 230) the position and orientation of 
the surgical instrument with respect to the target feature. The process projects (at 235) an 
indicia (e.g., a three-dimensional shape, points and/or lines) on the intraoperative image 
5 relating the position and orientation of the surgical instrument to the target spatial feature. 

Figures 3 and 4 illustrate exemplary user interfaces for the imaging systems that use 
the processes illustrated in Figures 1 and 2. Figure 3 shows an exemplary user interface (UI) 
for ultrasound-enhanced endoscopy. The left panel shows the endoscopic view with a 
superimposed targeting vector and a distance measurement. The right panels show 

10 reformatted cross-sectional planes through the acquired 3D ultrasound volume. Figure 4 
shows another UI for ultrasound-enhanced endoscopy. The left panel shows the endoscopic 
view with virtual tool tracking and visualization and vasculature acquired through Doppler 
imaging. The lower right panel shows volume-rendered 3D ultrasound. 

The UIs of Figures 3 and 4 support interactive rendering of the ultrasound data to 

15 allow a user to locate and mark the desired region of interest in the ultrasound image volume. 
The UIs allow the user to locate and mark target regions of interest. Hitting vascular 
structures is a serious hazard in endoscopic procedures. Visualization of the vasculature 
behind the surface tissue in the endoscopic view would assist in avoiding the vascular 
structures (anti-targeting). 

20 Figures 5 and 6 respectively illustrate exemplary surgical arrangements according to 

some embodiments of the invention. These systems, can: 

• track 500+ mm flexible endoscopes with an accuracy of 1.8 mm in position 
and 1 ° in orientation 

• acquire external 3D ultrasound images and process them for navigation in 
25 near real-time 



. allow dynamic targe, identification on any reformatted 3D ultrasound cross- 

sectional plane view. 

. optionally overlay dynamic Doppler ultrasound data, rendered using 

intensity based opacity filters, on the endoscopic view. 

. provide an overall coarse target accuracy of 10 mm, with a refined target 
accuracy of 5mm during breath-holds. 

* the system of Figure 5, a video source 500 (e.g., microscopic or camcorder),* used 
„ generate a video signal 501. In some embodiments described below, me video source 500 
U an endoscopic system. An intta-operative imaging system 502 (e.g., an ultrasonic system) 
captures an intta-operative imaging data stream 103. The information is displayed on an 

ultrasonic display 504. 

A trackable intta-operative imaging probe 505 is also deployed in one or more 
ttackable surgical tools 506. Outer tools include a trackable endoscope 507 or any 
intraoperative video source. The tracking device 508 has tracking wires 509 mat 
communicate a tracking data stteam 510. A navigation system 511 with a navigation 
Werfaco 512 is provided to allow the user to work with an intta-operative video image 513 
(perspective view). In the absence of video source this could be blank. 

Primary targeting markets 514 (pointing to a targe, outside the field of view) as well 
as secondary targeting markers 5 15 (pointing to a targe, inside the field of view) can be used. 
An intta-operative image 516 and an image of me lesion targe. 517 are shown with a virtual 
.epresemafionof surgical too.s or video source 518 (e.g., endoscope) as an orthographic view 
5,9 (outside view). Additions,*, an image over.ay 520 of any arbitrary 3D shape 
(anatomical representation or tool representation) can also be shown. 

Figmc 6 show, another exemplary surgical set-up. In Figure 6, several infrared vision 
cameras capture patient images. An ultrasonic probe positions an ultta-sound sensor in toe 
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patient. Surgical tools such as an endoscope are then positioned in the patient. The infrared 
vision cameras report the position of the sensors to a computer, which in turn forwards the 
collected information to a workstation that generates a 3D image reconstruction. The 
workstation also registers, manipulates the data and visualizes the patient data on a screen. 
5 The workstation also receives data from an ultrasound machine that captures 2D images of 
the patient. 

Since the geometry of a flexible endoscope in use changes continually, the field of 
view at the endoscope tip is not directly dependent on the position of a tracking device 
attached to some other part of the endoscope. This precludes direct optical or mechanical 
10 tracking; while useful and accurate, these systems require an uninhibited line of sight or an 
obtrusive mechanical linkage, and thus cannot be used when tracking a flexible device within 
the body. 

In order to make use of tracked endoscope video, six extrinsic parameters (position 

and orientation) and five intrinsic parameters (focal length, optical center co-ordinates, aspect 
15 ratio, and lens distortion coefficient) of the imaging system are required to determine the pose 

of the endoscope tip and its optical characteristics. The values of these parameters for any 

given configuration are initially unknown. 

A magnetic transducer is inserted into the working channel at the endoscope tip, 

positioning the field generator so that the optimal sensing volume encompasses the range of 
20 sensor positions. At this time, a 6 DOF miniaturized magnetic tracking system with metal 

insensitivity is used, although recent developments promise improved systems in the near 

future. 

In order to correctly insert acquired ultrasound images into the volume dataset, the 
world co-ordinates of each pixel in the image must be determined. This requires precise 
25 tracking of the ultrasound probe as well as calibration of the ultrasound image. Current 



calibration techniques are too cumbersome and time consuming to be performed prior to each 
use of the 3D ultrasound system. 

When tracking ultrasound data, the region of interest may be a significant distance 
from the probe itself. Consequently, any tracking error is magnified when the probe's 
5 orientation is projected to locate the region being imaged. 

One of the advantages of the ultrasound reconstruction engine is that it can be adapted 
to any existing ultrasound system configuration. In order to exploit this versatility, a simple 
and reliable tracking-sensor mount capability for a variety of types and sizes of ultrasound 
probes is used, as it is essential that the tracking sensor and ultrasound probe maintain a fixed 
1 0 position relative to each another after calibration. The surgeon may also wish to use the probe 
independently of the tracking system and its probe attachment. 

Accurate volume reconstruction from ultrasound images requires precise estimation 
of six extrinsic parameters (position and orientation) and any required intrinsic parameters 
such as scale. The calibration procedure should be not only accurate but also simple and 
15 quick, since it should be performed whenever the tracking sensor is mounted on the 
ultrasound probe or any of the relevant ultrasound imaging parameters, such as imaging 
depth or frequency of operation, are modified. An optical tracking system is used to measure 
the position and orientation of a tracking device that will be attached to the ultrasound probe. 
In order to make the system practical to use in a clinical environment, spatially calibration of 
20 the intrinsic and extrinsic parameters of the ultrasound probe is done. These parameters will 
then be used to properly transform the ultrasound image into the co-ordinate frame of the 
endoscope's field of view. 

The initial solution will be to use magnetic tracking for the ultrasound probe. An 
alternative solution is to track the probe using an optical tracking system. Tracking devices 
25 and a corresponding universal mounting bracket are deployed. In the typical 2D case, 

20 



acquired images are subject to scaling in the video generation and capture process. Since 
video output is not used, but the volumetric ultrasound data is accessed directly, this will not 
be an issue. The intrinsic parameters of the 3D probe, which will have been calibrated by the 
manufacturer, will be unmodified. A 200 _ 200 _ 200 mm phantom of tissue-mimicking 
material is used with an integrated CT-visible tracking device. Distributed along all three 
dimensions within the phantom will be cylinders and cubes, 20 mm in diameter and 
containing CT contrast material with modified acoustic impedance. The phantom will be 
imaged using the ultrasound probe; the transformation between the ultrasound volume and a 
previously acquired, reference CT volumetric image will be computed using intensity-based 
rigid registration (which requires the intensities of the two images to be similar in structure, 
but not in value). This transformation and the known position of the phantom's tracking 
device will be used to determine the relationship between the ultrasound imaging volume and 
the ultrasound probe's tracking device. Successful calibration requires an unchanged 
geometry. The phantom will be designed to withstand relocation and handling without 
deformation. A quick-release clamp attached to the phantom will hold the ultrasound probe 
during the calibration process. 

In order to locate and mark the desired region of interest in the ultrasound image 
volume, an interface supports interactive rendering of the ultrasound data. An interactive 
navigation system requires a way for the user to locate and mark target regions of interest. 
Respiration and other movements will cause the original location of any target to shift. If 
targets are not dynamically tracked, navigation information will degrade over time. The 
visibility of regular biopsy needles under ultrasound is poor and hitting vascular structures is 
a serious hazard in endoscopic procedures. Visualization of the vasculature behind the 
surface tissue in the endoscopic view would assist in avoiding it (anti-targeting), but 
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segmentation is a difficult, computationally intensive task. To address the foregoing, the 
navigation system operates in three distinct modes. 

The first is target identification mode. The imaged ultrasound volume will be 
displayed to allow the surgeon to locate one or more target regions of interest and mark them 

5 for targeting. The system will show an interactive volumetric rendering as well as up to three 
user positionable orthogonal cross-sectional planes for precise 2D location of the target. 

In the second mode, the endoscope will be used to set the position and orientation of 
the frame of reference. Based on these parameters and using the optical characteristics of the 
endoscope, the system will overlay target navigation data on the endoscope video. This will 

10 allow the surgeon to target regions of interest beyond the visual range of the endoscope's 
field of view. Displayed data will include the directions of, and distances to, the target 
regions relative to the endoscope tip, as well as a potential range of error in this data. 

The final mode will be used to perform the actual biopsy once the endoscope is in the 
correct position. The interactive imaged ultrasound volume and cross-sectional planes will be 

15 displayed, with the location of the endoscope and the trajectory through its tip projected onto 
each of the views. The endoscope needle itself will also be visible in the ultrasound displays. 

This will help to position the biopsy needle in the center of the lesion without being 
limited to a single, fixed 2D ultrasound plane emanating from the endoscope tip, as is 
currently the case. (That 2D view capability will however be duplicated by optionally 

20 aligning a cross-sectional ultrasound plane with the endoscope.) In the first implementation 
of the flexible endoscope tracking system, the magnetic sensor will need to be removed from 
the working channel in order to perform the biopsy, and the navigation display will use the 
stored position observed immediately before its removal. Ultimately, though, a sensor will be 
integrated into the needle assembly, which will be in place at calibration. 
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When a targe, is identified, «he Region surrounding «he ,arget in «he ultrasound 
volu me wifi * acred as a reference, .ogemer whh fire tracted orientafion of fire vo,ume. A 
.bregion of each s occessive,y-ac,uired uldasound voiunre, centered a, dre target poaiticn in 
ft. preceding vo.ume, will be re-samp,ed using the crientancn of tne reference targe, 
region. Three-dimensional eross-corre,a«io„ of the re-aatnp.ed subregion widt the 
reference subregion will be used to find the new location of fire target 

THia agamic „** wUl follow eacb .arge, over time; if the ayatem is displaying 

target relative to the endoscope. 

Vaacdar a— return a snong, we« different Dopp.er aigna,. The dynamic 
^ound da. may be rendered in real time uaing intensity-based opacity filters, mafcing 
ocular avenues Kansparen,. This *** — *° vasctdar aducmre without 
^rmg cumpumttonally-demanding deformabie geometric mode,a for segmentation, dma 
b eing ab,e to fodow movement and deformation in real time, .f dre ,ag is aiguificant, 
15 navigadon accuracy w,„ be degraded when the target move, Where optima, accuracy s 

required. 

L ens dis to rdou compensadon is performed for dte da* display in rea, dme, so ma, dte 
su perimposednaviga,iondisp 1 ay maps accuracy » the underlying endoscope video. 

A new uhrasound volume wiU replace the nex, moa, recen, volume in Us enbre*. 
much aa 1, does on ,he diap.ay o, due uhraaound machine Use* ahhough possibly a, a 
„ spada. .ocadon. This avoids many problemadc areaa such as mis.eading old dam, 
da ,a expiranon, unbounded imaging volumes, - locking rendering da., hts^d, a simple 
pbrg-pong buffer pair may be used ; one may be used for navigadon and display whde dr. 
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other is being updated. Another benefit of this approach is that the reduced computational 
complexity contributes to better interactive performance and a smaller memory footprint. 

All phantoms will be manufactured to tolerances at least 40 times smaller than the 
expected system error for the test associated with the phantom. This degree of inaccuracy is 
5 small enough to be included in the total system error without any significant impact on 
specifications. 

Computerized object recognition and surveillance are used in order to overlay images 
of the operative field from ultrasound onto a 3D image of the object using the Laser 
Targeting System and internal anatomical markers. The 3D images will be created in the 

10 workstation both from high resolution MR and CT images and intra-operative ultrasound, 
obtained with volume acquisition techniques. The system enables an interactive and 3D 
guidance system for surgeons with maximal flexibility and accuracy. With these techniques, 
surgery will be performed with the same tools and basic procedures as with non-guided 
operations, yet the precision and minimization of trauma provided by frame-based stereotaxy. 

15 An exemplary operation of the system will be discussed next help to delineate its 

features. Take the case of a deep intra-axial brain lesion for which resection is planned. Prior 
to obtaining the pre-operative images, 4 markers are placed on the patient. These can be small 
(-2mm), flashing diodes surrounded by pantopaque filled spheres glued to the skin. 
Pantopaque is an oil-based, iodine containing X-ray contrast agent that, until recently, was 

20 used for myelograms. The iodine makes it visible on CT images while the oil base renders it 
visible on MRI examinations. 

Depending on the imaging characteristics of the lesion and the important surrounding 
anatomy, (i.e., whether there are calcifications, whether it enhances with gadolinium, etc.) CT 
or MR scans or both will be performed. Typically high resolution contrast enhanced MR 

25 images and MR angiograms would be obtained. The image data would be transferred to the 
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workstation (via the hospital's computer network), volume rendered, and (in the case of 
multiple imaging modalities) fused. 

The image data will be segmented to allow detailed visualization in appropriate 
anatomic context of the lesion selected intra- and extra-axial structures, and the fiducials. 

5 Segmentation of the fiducial markers, brain, the vascular system and surface of the scalp 
would be fully automatic. The segmentation of the lesion, however, would be only partly 
automatic, as the irregular anatomy surrounding such lesions is currently too unpredictable 
for automatic segmentation algorithms. 

In the operating room, the patient would be positioned in the usual fashion. The 

10 optical tracking system would be positioned above and to the side of the patient's head. 
Chroma key techniques will automatically identify the flashing markers, enabling automatic 
and continuous registration and overlay of the patient's physical anatomy with the 3D image 
data sets. Chroma key is a video special effect technique that allows unique detection of 
flashing objects with known frequencies in 3D space (e.g., flashing light emitting diodes 

15 attached to the patient's head). Diode markers can also be added to conventional ultrasound 
probes and surgical tools (e.g., probes, scalpels) for their tracking in the stereotactic space. 
Using chroma key techniques, the markers are automatically recognized and overlaid on the 
display of the registered 3D image. 

A triangle formation of three markers on the ultrasound probe allows the tracking of 

20 its movement as it scans the surgical site, thus providing the system with volumetric 
ultrasound images. Continuous intra-operative registration of the patient's anatomy with the 
intra-operative ultrasound images is very important due to the movement and deformation of 
the brain tissue during the surgery. Intra-operatively acquired 3D ultrasound images are then 
being fused with the pre-operative CT or MR imagery using anatomical features visible to 

25 both modalities, such as vascular structures and the lesion. In addition, an extrapolated line 
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sending ftom fte Splayed image of a surgica. device, indicates the trajectory of the 
p,anned approach. Moving the too, automatically leads to a change in the dispiayed potential 
rectory, and the iocation of the L.E.D. on the too, enah.es determination of the prec.se 
dep ft and location of rhe «oo., .hereby enah.ing precise de.ennina.ion of fte depft and 
5 focation of fte operate site. Th^ore. mia system no. only simptifies fte planning of a 
utinimally invasion approach to a direc. and interactive task, bnt i. is a.so more precise than 
*. conventional systems dne * its innn.pera.ive image update* and registration nsing 
ultrasound imagery. 

Furthermore, me information provided hy fte video camera-hased objec. recognition 
10 system, fte iaser feting system w«I mrfter aid in .ocalization of fte surgical site, ftns 
leasing fte registration and image overiay per— and accuracy hy locals fte 
are a for which re-registration U needed. Tbia information allows fte worsen ,o 

(and if desired overlaid wift) fte approach. The 3D reformatting uses volume display 
U technic and aUows —eons variation o, transpamncy. Wift ftia technic deep as 
wdl as superficial shucftrea, can be seen in come*, fterehy considerably enhancing intra- 
operative guidance. 

T* system software has «wo aspect one dealing wift enhancement .o fte user 
iu.erft.ee, and fte second focusing on a,goriftms for image manipulation and region. 

• , f tv,„ means for image segmentation, volumetric visuahzation, 
20 These algorithms consist of fte means lor image 

taage fusion and image overlay. In one embodiment, fte system provides: 

i) A "user friendly" interface, to ft.cili.ate usage in fte operating room. 
U) interactive image analysis and manipulation routines (e.g. arbitrary cuts, 
i^e segmentation, image magnification and transformation) on fte worsen system. 
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iii) Seamless Interface between the optical tracking system (provided with a diode 
encoded pointer) with a computer workstation. 

iv) Seamless interface of a video camera and the laser targeting system with the 
optical tracking system and workstation. 

5 v) Overlay of video images from (iv) on the 3D data from (iii) using diode 

markers positioned on the surface of the test objects. Test the accuracy of the pointer's 
guidance in objects. 

vi) Inclusion of an ultrasound probe with the optical tracking system and 
workstation in order to obtain 3D ultrasound images. 
10 vii) Fusion of 3D ultrasound images from (vi) to the 3D MR/CT images from (iii) 

using diode markers positioned on the ultrasound probe. 

viii) Deform test objects after their scan. Attempt to correct the deformation with 
3D ultrasound images using linear displacement of the region of interest, then test the 
accuracy of the pointer's guidance in test objects. Test objects increase in complexity as 

15 testing proceeds. 

The above-described medical systems have numerous advantages. For instance, these 
systems enhance intra-operative orientation and exposure in endoscopy, which, in turn, 
increases surgical precision and speeds convalescence and thereby reduces overall costs. The 
ultrasound-enhanced endoscopy (USEE) improves localization of targets (e.g., peri-lumenal 

20 lesions) that lie hidden beyond endoscopic views. On a single endoscopic view, some of 
these systems dynamically superimpose directional and targeting information calculated from 
intra-operative ultrasonic images. Magnetic tracking and 3D ultrasound technologies are used 
in conjunction with dynamic 3D/video calibration and registration algorithms for precise 
endoscopic targeting. With USEE, clinicians use the same tools and basic procedures as for 

25 current endoscopic operations, but with a higher probability of accurate biopsy, and an 
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tocre ased chance for the — — ^ "~" *" 

navigation. The sysfcms provide effeotive caUbration and «— of in- 
operative volumetric imaging data with video endoscopy images. 

Some of these systems acquire externa, 3D ultrasound images and process them for 
, navigation in near reaHime. These system aUow dynamic targe, identification on any 
reformatted 3D ultrasound cross-secnona, p.ane. The sys te m can automatical tiaclc the 
movement of me target as tissne moves or deforms dnring the procednre. These sy*ems can 
dynamical* map me target iocation onto me endoscopic view in form of a direction vector 
mi display quantifiable dam such as distance to target Optionally, me systems can provrde 
10 meting information on me dynamic 3D ultrasound view. The systems can virtually 
dualize the position and orientation of tracted snrgica, too,s in me uhrasoond view, and 
options,* .so in the endoscopic view. These systems aiao can overiay dynamic Doppier 
ultraS ound da., rendered using intensity based opacity filters, on me endoscopic view. 

The invention has been describe* in terms of specific examples, which are illustrative 
15 on,y and are no, to be constiued as limiting. The invention may be imp,— in dig*. 

Apparatus of the invention may be implemented in a computer program product tangibly 
bodied m a machine-readable storage device for execution by a computer processor; and 
m ethod steps of the invention may be performed by a computer processor executing a 
2 „ program » perform Amotions of the invention by operating on input da. and generating 
output Suitable processors inCude, by way of exampic, both genera, and specia, purpose 
coprocessors. Storage devices suitab,e for tangib.y embodying computer program 
actions include aU forms of non-vo.attie memory inCuding, but no. timited to: 
semi— r memory devices such as EPROM, EEPROM, and flash devices; magnetic 
25 disks (fixed, floppy, and removable); outer magnetic media such as tape; optica, media such 
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as CD-ROM disks; and magneto-optic devices. Any of.be foregoing may be supplemented 
by , or incorporated in, speciaUy-design*. app.ication-spec.fic integral circuit (ASICs) or 
suitably programmed field programmable gate arrays (FPGAs). 

Fr „m me a foregoing disclosure and certain variations and modifications aheady 
Closed .herein for pmposea of illuatiation, it will be evident to one skilled in the re.evam 
^ that me present inventive concept can be embodied in forms different fiom moae 
ascribed and i, wffl be understood that me invention is intended to extend ,o such nnfirer 
variations. Wbile me prefemed forms of.be invention bave been sbown in the drawings and 

md Scribed since variations of me preferred forms win be apparent to those skiued in me 
^^usmescopeofmemventionisdennedbymefonovvmgciaimsandmeir^iva.en.s. 
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